AI-Based Clinical Decision Support Systems in Primary Care: Systematic Review of Clinical Implementation

Introduction

Primary care is central to high-performing health systems, contributing to improved population health outcomes, greater equity, and reduced avoidable hospitalizations [1-4]. Yet, primary care is increasingly strained [5], with rising case complexity, workforce shortages, and fragmented information systems placing increasing pressure on clinicians [6-9]. At the same time, the volume of clinical evidence continues to expand at a pace that exceeds clinicians’ capacity to remain aligned with current best practices [10]. These challenges have accelerated interest in digital tools to support clinical decision-making and workflow in primary care settings [11,12].

Clinical decision support systems (CDSSs) can assist clinicians by generating patient-specific recommendations at the point of care [13]. Meta-analyses of controlled trials indicate that traditional CDSSs achieve small to moderate improvements in targeted processes of care (eg, prescribing, test ordering, documentation, and vaccination) [14]. However, these effects do not consistently translate into better clinical outcomes, with evidence indicating small and inconsistent improvements at the patient level [14]. This gap highlights a key limitation of traditional CDSSs, as they have limited capacity to account for variability in patient characteristics and clinical contexts [15,16]. AI methods can be integrated into CDSSs to improve data-driven precision and enable more individualized recommendations [15,16].

Most evidence on AI-based CDSSs (AI-CDSSs) remains concentrated in the early stages of the evaluation pathway [17-19]. Most AI-CDSS studies focus on model development and validation, often reporting strong technical performance [20-24]. Performance evaluations are typically conducted using curated datasets and predefined tasks that do not reflect the complexity and variability of routine clinical care [6,25,26]. Performance assessments conducted in closed contexts therefore poorly predict how AI-CDSSs will perform when deployed in clinical practice.

To be effective, AI-CDSSs must align with clinical workflows and support, rather than disrupt, the work of clinicians [7,27]. When there is a misalignment, decision support can introduce additional clinical or administrative steps into already time-constrained consultations or generate recommendations that are difficult to implement [28]. Clinical misalignments commonly lead to nonadoption, inappropriate use, and long-term abandonment of AI-CDSSs [7,29-31]. Even systems with strong technical performance have failed to produce clinical impact because they were not effectively integrated into clinical practice [32].

AI-CDSSs influence clinical decision-making in ways that shape diagnosis and management, raising clinical safety considerations [33]. Errors in predictions or recommendations, if not identified or appropriately contextualized, can lead to inappropriate or delayed care [34,35]. How clinicians interact with AI-CDSSs, such as relying on recommendations without sufficient critical appraisal or disregarding relevant outputs, can shape how these risks materialize in practice [36]. Data from AI incident databases document thousands of AI-related incidents and hazards reported over the past decade, emphasizing the scale of these concerns [37]. However, the processes through which risks are identified, monitored, and mitigated in clinical settings remain insufficiently characterized [37-39], limiting the ability to determine whether AI-CDSSs can be safely integrated into care.

The rapid development, increasing availability, and commercialization of AI-CDSSs are accelerating their clinical use [40,41]. This pace of deployment could exceed the availability of effectiveness and safety evidence in clinical settings [19,42-47]. When evaluation does not keep pace with deployment, systems are introduced into care without adequate evaluation in routine practice [45,48,49], limiting impact assessment on clinical decision-making and health system oversight [43,50]. Previous reviews have mapped AI use in primary care [24,51] and highlighted barriers to AI-CDSS implementation, including workflow integration and feasibility [52,53], but have not systematically synthesized whether AI-CDSSs improve outcomes or how safe they are compared with usual care in routine primary care settings. This evidence gap aligns with recent calls for clinical evaluation of AI-CDSSs to support implementation decisions in health care [41,44,45,54,55]. The aim of this review was to synthesize evidence on the effectiveness and safety of AI-CDSSs compared with usual care or non–AI-based systems in primary care settings.

Methods

Study Design

We conducted a systematic review following the Cochrane Handbook for Systematic Reviews of Interventions [56] and reported the results according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines [57]. The completed PRISMA 2020 checklist is provided in Checklist 1. We registered the protocol in PROSPERO [58]. Following protocol registration, departures from the registered methods were limited and are summarized in Multimedia Appendix 1. These consisted of planned analyses that were not undertaken because the characteristics of the included evidence did not support their meaningful application (Risk of Bias due to Missing Evidence and Grading of Recommendations Assessment, Development, and Evaluation [GRADE]), and 1 additional exploratory analysis introduced post hoc to facilitate interpretation of heterogeneity by classifying implementation contexts using an operationalized approach informed by the Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework.

Eligibility Criteria

We present the eligibility criteria according to the population, intervention, comparator, and outcomes (PICO) framework [59] in Table 1.

Information Sources and Search Strategy

We developed a search strategy in collaboration with an experienced librarian (FB) and reported it in accordance with the PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses literature search extension) guidelines [60] (Checklist 2). We tested the sensitivity of the strategy [61] using studies identified in a previous scoping review [51] that were expected to be retrieved by our strategy. The search was not updated after the initial sensitivity testing. We provide the full electronic search strategies for all databases in Multimedia Appendix 2.

We searched the following databases on June 27, 2025, without validated filters or date and language restrictions: MEDLINE (Ovid), Embase (Elsevier), CINAHL (EBSCOhost), Web of Science Core Collection (Clarivate), and the CENTRAL. We combined database-specific controlled vocabulary (eg, MeSH and Emtree) and free-text terms related to AI and CDSS with terms capturing study designs relevant to effectiveness and safety (ie, randomized, quasi-experimental, and observational studies). We did not search for gray literature sources (eg, conference proceedings and institutional reports) because of the focus on peer-reviewed evidence. However, we searched the trial registries of included studies to identify ongoing or unpublished results. We conducted backward and forward citation searching [62] of all included studies and relevant reviews identified during the selection process, using Scopus on August 6, 2025.

All retrieved records were imported into the management software Covidence (Veritas Health Innovation), where duplicates were removed using its automated function.

Selection and Data Collection Processes

We conducted calibration training on eligibility criteria interpretation among reviewers (FN, MS, and SO) using the first 100 references identified. If interrater agreement (percent agreement) was below 80% [63], we clarified the eligibility criteria through discussion and a think-aloud approach [64] and conducted a new training. Pairs of reviewers (FN, MS, and SO) applied the eligibility criteria to independently screen the titles and abstracts of all potentially eligible citations and then to full-text articles retrieved. We resolved discrepancies by team consensus. We documented reasons for full-text exclusion and reported them using the PRISMA 2020 flow diagram.

Data Items

We developed a standardized data extraction form informed by existing reporting guidelines, including DECIDE-AI for early-stage clinical evaluation of AI-CDSSs [65], CONSORT-AI (Consolidated Standards of Reporting Trials–AI) for clinical trials of AI interventions [66], and the APPRAISE-AI tool for quantitative evaluation of AI-CDSS studies [67]. Given the small number of eligible studies, the form was pilot tested on the most detailed study [68]. Two independent reviewers (FN and SO) performed the data extraction, and disagreements were resolved by a third reviewer (MS). Extracted variables covered study characteristics (eg, study period and number of participating institutions), participant characteristics (eg, sample size and baseline features), and intervention details (eg, purpose and outputs). We also collected data on outcomes and effects (eg, measurement instruments and between-group comparisons) and safety (eg, risk mitigation strategies and reported harms or unintended effects).

Risk of Bias Assessment

We assessed individually randomized and cluster-randomized trials using the corresponding versions of Risk of Bias-2 [69] and quasi-experimental studies using ROBINS-I (version 2) [70]. Two reviewers (FN and SO) independently evaluated the risk of bias for the main outcome of each study, and disagreements were resolved by team consensus.

Synthesis Method

Owing to substantial heterogeneity across studies, we did not conduct a meta-analysis [71,72]. First, there was limited evidence for any given comparison, with most outcomes reported in only 1 or 2 studies, resulting in minimal overlap of comparable effect estimates [72]. Second, studies exhibited marked clinical and methodological diversity, including variation in target conditions, AI-CDSS functions, implementation contexts, study designs, and outcome definitions [72]. Under these conditions, the assumption of a common underlying effect was not met, and quantitative pooling would risk producing misleading summary estimates. This approach is consistent with methodological guidance from the Cochrane Handbook [72].

We therefore performed a narrative synthesis and reported it in accordance with the Synthesis Without Meta-analysis (SWiM) reporting guidelines [71]. We categorized outcomes into 3 levels: patient level, referring to measures affecting patients, such as symptom improvement or function; clinician level, including outcomes related to provider knowledge, behavior, or workload; and system level, encompassing organizational or economic outcomes, such as health care utilization or cost-effectiveness. We synthesized findings using the direction of effects (improvement, no effect, or worsening), which was determined based on the statistical results reported in the original studies (eg, P values or equivalent information). All included studies were considered, and findings were interpreted using study design and risk of bias. We presented findings using structured tables and narrative summaries. We provide the completed SWiM checklist in Checklist 3.

We explored heterogeneity through structured comparison across implementation contexts. To describe the implementation context, we classified each study as simple, complicated, or complex using an operationalized approach informed by selected items from the NASSS-Complexity Assessment Tool (CAT) short version [73,74]. As NASSS-CAT was developed to support prospective assessment of technology projects and includes domains not consistently reported in published studies, we adapted relevant items into 4 observable criteria derived from the extracted data: condition complexity, decision logic, system complexity, and preference sensitivity. Each criterion was coded using predefined definitions (Multimedia Appendix 3). Two reviewers (FN and MS) independently classified context as simple when all criteria indicated low complexity and as complex when at least two criteria indicated high complexity. All other scenarios were classified as complicated.

We classified safety information according to the DECIDE-AI reporting guidelines [65] across 3 domains: patient safety (prespecification and reported harms or unintended effects), AI system (risk mitigation and significant errors or malfunctions), and clinician interaction (agreement and override).

For AI-CDSS functions reporting, we used the taxonomy of front-end clinical decision support tools [75]. We then complemented this classification by extracting the corresponding operational functions, defined as the concrete actions performed by the system within the study context (eg, alert optimization, monitoring and follow-up, treatment optimization, and documentation or care summary).

Results

Study Selection

We retrieved 4074 records from the search. After removing duplicates, 3034 (74.5%) records remained for the selection. Of these records, 126 (4.2%) were retained for full-text screening. We identified 12 supplemental records from handsearching and trial registration checking. At the end of the process, we included 14 records on 11 distinct studies (we present the mapping of included records to unique studies in Multimedia Appendix 4). We present a flow diagram of the selection process in Figure 1.

‎

Characteristics of the Studies

We present a summary of the characteristics of the included studies in Table 2, and a detailed version is available in Multimedia Appendix 5.

We identified 11 included studies published between 2019 and 2025 and conducted across North America (n=7, 63.6%) and Europe (n=4, 36.4%). Four (36.4%) studies used experimental designs, and 7 (63.6%) were quasi-experimental, applying a variety of comparison strategies, including concurrent control groups, historical or matched controls, and before-after or self-controlled analyses. Study durations ranged from approximately 1 to 12 months. According to the DECIDE-AI reporting items, none of the studies reported patient involvement in the evaluation of the interventions. Five (45.5%) studies reported at least one conflict of interest, and 5 (45.5%) were funded by industry.

Risk of Bias Assessment

We evaluated the overall risk of bias as high for all experimental studies. The main concern for the 2 parallel randomized controlled trials (RCTs) was related to deviations from the intended interventions. For the 2 cluster RCTs, the main concerns were linked to the identification and timing of recruitment and the measurement of the outcomes.

Among quasi-experimental studies, 1 was at serious risk of bias, while all others were at critical risk of bias. Five critical-level assessments were derived from the ROBINS-I screening stage (B1-B3), reflecting concerns related to uncontrolled confounding and/or outcome measurement. In the remaining study rated at critical risk of bias, concerns were related to missing data. We provide overall judgment and detailed domain-level assessments in Multimedia Appendix 6.

Characteristics of the Participants

We present a summary of the characteristics of the participants included in the studies in Table 3, and a detailed version is presented in Multimedia Appendix 7.

AI-CDSSs were primarily evaluated in musculoskeletal conditions (3/11, 27.3%), urinary tract infections (2/11, 18.2%), and diabetes (2/11, 18.2%), with 1 (9.1%) study addressing multiple conditions. Of the 11 studies, 6 (54.5%) reported clinician sample sizes, ranging from 7 to 358. Among these 6 studies, 4 (66.7%) did not report any clinician characteristics. Age and sex were reported in 2 (33.3%) studies, and ethnicity in 1 (16.7%) study. Of the 11 studies, 8 (72.7%) reported patient sample sizes, ranging from 14 to 24,083 in the experimental groups and from 14 to 12,103 in the control groups. Among these 8 studies, age was reported in all (100%), sex and ethnicity in 5 (62.5%), and gender in 4 (50%). When reported (8/11, 72.7%), the number of participating institutions varied from 1 to 45 and included urban, rural, or mixed health care settings.

Characteristics of the Interventions

We describe the control and experimental interventions in Table 4.

Of the 11 studies, control interventions were mainly usual primary care practice (n=10, 90.9%). The remaining study had a digital care program as the control group. One study used a rule-based CDSS as the control intervention arm. Experimental interventions generally preserved the same clinical care as the control while adding an AI-CDSS.

We present the identified AI-CDSS in Table 5.

Studies evaluated 11 distinct AI-CDSSs with different purposes. Seven (63.6%) AI-CDSSs aimed to support condition management, 3 (27.3%) aimed at the early detection of a condition or a deterioration, and 1 (9.1%) aimed to reduce alert fatigue. Preimplementation training occurred in 3 (27.3%) studies, and implementation support occurred in 2 (18.2%) studies.

We provide AI characteristics of the identified AI-CDSS in Table 6.

Six (54.5%) identified AI-CDSSs were commercial products, 2 (18.2%) were institutionally developed but not open source, and 3 (27.3%) did not report the accessibility of the underlying AI method. Nine (81.8%) studies reported the type of AI system; among these, 4 (44.4%) used machine learning, and 3 (33.3%) used deep learning approaches. No study reported when the AI model was developed. Regarding interoperability, 4 (36.4%) studies evaluated stand-alone systems, 2 (18.2%) reported electronic health record (EHR)–embedded systems, and 3 (27.3%) used EHR data without specifying integration within the EHR interface. Three studies (27.3%) did not report interoperability information.

We present the CDSS characteristics in Table 7.

Expert system functions and relevant information display were each reported in 8 (72.7%) studies, and point-of-care alerts or reminders in 4 (36.4%). Eight (72.7%) of the 11 AI-CDSSs included multiple functions. These functions translated into operational capabilities, such as treatment optimization (7/11, 63.6%), risk prediction (5/11, 45.5%), monitoring or follow-up (4/11, 36.4%), documentation or care summary (4/11, 36.4%), screening (3/11, 27.3%), referral (2/11, 18.2%), and alert optimization (2/11, 18.2%). Modes of delivery included dashboards (5/11, 45.5%), alert-based systems (3/11, 27.3%), printed or periodic reports (2/11, 18.2%), and email notifications (1/11, 9.1%). One (9.1%) study did not report the mode of delivery. All systems aimed to support rather than replace clinician decision-making, as explicitly stated in 3 (27.3%) studies and implicit in the remainder. In 7 (63.6%) studies, AI-CDSS use required additional clinician tasks, including access to and review of dashboards (5/11, 45.5%), manual data entry (1/11, 9.1%), acquisition of clinical images (1/11, 9.1%), printing of reports (1/11, 9.1%), and review of system-generated reports (1/11, 9.1%).

Using the classification approach adapted from the NASSS framework, included studies were assigned to simple (4/11, 36.4%), complicated (4/11, 36.4%), and complex (3/11, 27.2%) implementation contexts (see Table 8 and Multimedia Appendix 3 for rating details).

Outcomes and Effects

Overview

We present a summary of between-group effects in Table 9 and provide a detailed table, including within-group results, in Multimedia Appendix 8.

Among the 11 included studies, outcomes were most frequently reported at the clinician level (n=8, 72.7%), followed by the system level (n=6, 54.5%) and the patient level (n=5, 45.4%). Nearly half (n=5, 45.4%) of the studies assessed outcomes across all 3 levels.

Clinician-Level Outcomes

Four (50%) studies reported workflow outcomes assessed during the implementation period, resulting in mixed effects. Two experimental studies reported reductions in alert burden and clinician workload across implementation periods ranging from short (~2 mo) to longer periods (~12 mo). These studies were conducted using protocolized or structured decision logic. Two quasi-experimental studies reported no effect on interacted alert counts in a simple context with an automated EHR-embedded AI-CDSS for alert optimization, and no effect on appointment length in a complex context characterized by individualized decision logic, a multicomponent system, and high preference sensitivity.

Four (50%) studies reported decision behavior outcomes during the implementation period. One experimental and 3 quasi-experimental studies examined prescribing behavior, guideline adherence, or order count with heterogeneous effects across and within studies. These studies were conducted across different implementation contexts. In the simple context, alert suppression was not associated with changes in the order count outcome. Two studies conducted in complicated contexts reported mixed effects on prescribing behavior and positive or neutral effects on adherence to clinical guidelines. The experimental study conducted in a complex context reported increased use of treatment modalities. These complicated and complex contexts were characterized by clinical heterogeneity, structured or individualized decision logic, and moderate to high preference sensitivity.

One quasi-experimental study (n=1, 12.5%) context evaluated attitudinal outcomes at the end of the short implementation period (~1 mo) in a complicated context, reporting improvements in care coordination and patient preparation, alongside concerns regarding the usefulness of the system and its impact on clinical work.

Patient-Level Outcomes

Two (40%) studies reported patient-reported outcomes assessed at 12 weeks in musculoskeletal conditions. Pain intensity and mental health were the only outcomes assessed in more than 1 study. Across these studies, findings for pain intensity were inconsistent, with 1 experimental study reporting no effect and 1 quasi-experimental study reporting a negative effect. Both studies reported no effect on mental health outcomes. These studies were conducted in complex contexts characterized by clinical heterogeneity, individualized decision logic, multicomponent systems, and high preference sensitivity.

Two (40%) studies reported clinical outcomes assessed during or after the study period. One experimental study reported no effect on asthma occurrence and control, while 1 quasi-experimental study observed a positive effect on treatment success. Both studies were conducted in complicated contexts characterized by clinically heterogeneous but stratified conditions, protocolized decision logic, and moderate preference sensitivity, and differed in system complexity.

Two (40%) studies reported process-of-care outcomes with mixed effects. One experimental study reported a positive effect on the detection rate of low ejection fraction in a simple implementation context characterized by a well-defined diagnostic target, protocolized decision logic, an operationally simple system, and low preference sensitivity. An experimental study reported no effect on medication use in a complex context involving clinical heterogeneity, individualized decision logic, a multicomponent system, and high preference sensitivity.

One quasi-experimental study (1/5, 20%) reported a patient-reported experience outcome (satisfaction with the program) at 12 weeks with a positive effect.

System-Level Outcomes

Three (50%) studies evaluated service delivery outcomes during the implementation period and reported mixed results. One quasi-experimental study reported improvements in program completion and patient engagement (eg, reach-outs and messages), while another quasi-experimental study found increased attendance at follow-up appointments. These effects were observed across different implementation contexts, including both complex and simple contexts. Two studies reported neutral or negative effects on service delivery indicators, such as the number of consultations (experimental study) and sessions performed per week (quasi-experimental study), both conducted in complex contexts.

Two (33.3%) experimental studies evaluated economic outcomes with mixed effects. One study reported that the intervention was cost-effective, while the other found no effect on health care costs. The 2 studies were conducted in different implementation contexts, including 1 simple context involving a well-defined diagnostic target with low preference sensitivity and 1 complicated context with a heterogeneous condition and moderate preference sensitivity.

Two (33.3%) studies reported utilization outcomes with positive effects. An experimental study reported an increased use of diagnostic procedures among high-risk patients but no clinical effect difference in the intervention sample. This result was observed in a simple context characterized by a well-defined diagnostic target and a protocolized decision logic. One quasi-experimental study reported improved physiotherapist-to-patient ratios in a complex context.

One (16.7%) quasi-experimental study reported no impact on safety during the implementation period in a complex context.

Safety Evaluation

We present the reporting of safety-related information in Table 10. Two (18.2%) studies prespecified structured monitoring of adverse events, primarily relying on patient-reported side effects or standardized questionnaires rather than active surveillance of AI-related harms. Across these 2 studies, as well as 1 additional study reporting post hoc harms and unintended effects, no severe adverse events attributable to the AI-CDSS were identified. Risk mitigation strategies were described in 4 (36.4%) studies and included human-in-the-loop oversight, internal quality checks, and automated safeguards such as validation of severe alerts or continuous retraining with monitoring dashboards. Reports of errors or malfunctions were uncommon (n=3, 27.3%) and mainly incidental, involving image quality issues, software glitches, or technical exclusions of participants. No study reported clinician agreement with or override of AI-generated recommendations.

Discussion

Principal Findings

This systematic review synthesized evidence on the effectiveness and safety of AI-CDSSs in primary care across 11 studies. Two RCTs, conducted in a simple implementation context with screening or alert optimization AI-CDSSs, reported positive effects on workflow, economic, and utilization outcomes. In contrast, studies conducted in more complex contexts reported limited or inconsistent effects, with patient-level outcomes rarely improved. Across studies, most systems were deployed as stand-alone tools requiring additional clinician effort, and follow-up periods were generally short, reflecting early phases of implementation. Safety was not evaluated as an integrated, multilevel construct. Only 2 studies prespecified patient safety outcomes. System-level safeguards were described but not evaluated, and no study examined how clinicians engaged with AI-generated recommendations. The current evidence base does not clarify how system performance, clinician behavior, and patient outcomes interact to shape safety in practice. The overall strength of evidence is further limited by substantial heterogeneity and consistently high or critical risk of bias, restricting confidence in the observed effects and their applicability across clinical contexts.

Effectiveness Considerations

Positive effects of AI-CDSS implementation were reported more consistently when clinical decisions followed structured and protocolized action pathways. In our review, the most consistent effects emerged in studies targeting screening, referral, or alert optimization, where clinical rules were explicit and thresholds were predefined. In these contexts, recommendations could be readily integrated into routine workflows and translated into immediate clinical responses. Prior CDSS research similarly suggests that decision support is more likely to improve care when recommendations are delivered at the point of care, embedded within clinical workflows, and linked to clear actions [89-91]. However, structured decision contexts alone may be insufficient to ensure effectiveness, as systems that increase workflow burden, generate excessive interruptions, or poorly align with local practices may reduce clinician responsiveness over time [92-96]. Alert fatigue represents a well-documented failure mode, with studies reporting override rates exceeding 50% to 73% for medication-related alerts, with clinicians becoming progressively less likely to accept alerts as their frequency increases [94,97,98]. This suggests that the effects of AI-CDSSs in structured contexts may depend not only on the clarity of the underlying clinical rules but also on the degree to which system recommendations remain actionable, relevant, and aligned with the realities of clinical practice.

In settings requiring complex or individualized decision-making, AI-CDSS effects on clinical outcomes appear less consistent, potentially not because the systems fail, but because they are used differently in ways that leave a limited measurable trace. Clinicians integrating evolving clinical information, patient goals, and preference-sensitive trade-offs across longitudinal care trajectories face decisions that are less amenable to reduction into a single rule, threshold, or predefined action pathway [99-101]. Several mechanisms may contribute to this pattern. First, as decision tasks become more cognitively demanding, clinicians appear less likely to consult decision support systems and may shift toward situation-assessment tools rather than algorithmic recommendations [102]. Second, physicians tend to remain anchored to pre-existing diagnoses when using diagnostic CDSSs. Berner et al [103] found that system outputs more often reinforced than redirected clinical reasoning, a pattern consistent with confirmation and anchoring biases previously described in clinical reasoning research, particularly under cognitively demanding conditions [104,105]. Third, even when systems are consulted, clinicians may resist outputs perceived as undermining their decision autonomy or conflicting with the clinical context, particularly when system reasoning is opaque or inconsistent [106,107]. These mechanisms may shift AI-CDSSs from a directive toward a more interpretive or confirmatory role. In our review, a study targeting chronic musculoskeletal pain found that clinicians used AI-CDSSs primarily to support existing practice rather than to modify management decisions [68]. This pattern is consistent with broader evidence showing that CDSS effects on clinical outcomes in chronic disease are often modest and heterogeneous even when improvements in process measures are observed [108,109]. If AI-CDSS outputs primarily validate or refine reasoning rather than consistently triggering management changes, measurable downstream effects on clinical outcomes may be expected to remain small, variable, and difficult to detect. Decision complexity should be explicitly addressed during AI-CDSS design, implementation, and evaluation [29]. Future research may also need evaluation approaches that better capture interpretive and confirmatory influences on clinical reasoning rather than relying exclusively on downstream outcome measures.

Implementation success also depends on alignment with organizational context, clinical workflows, and clinician routines [7,53,110]. Many AI-CDSS failures appear to be sociotechnical rather than algorithmic [19,29,111-113]. In our review, most AI-CDSSs were implemented as stand-alone systems requiring additional clinician effort, including manual data entry, dashboard consultation, or review of system-generated outputs outside existing workflows. In primary care, increased workload consistently emerges as a major barrier to CDSS adoption, with human and organizational factors negatively influencing uptake regardless of technological performance [7]. Workflow integration, usability, and health care system–level factors have also been identified as persistent implementation challenges across clinical domains and CDSS types [53,110], reinforcing the importance of aligning decision support systems with routine clinical work [29,32]. These barriers may become particularly consequential in more complex care settings, where workflows are often less standardized. In these contexts, clinicians must continuously integrate evolving clinical and contextual information across encounters, providers, and competing patient priorities, as in chronic disease management, multimorbidity, or preference-sensitive care [52]. EHR integration is frequently proposed as a strategy to reduce these implementation barriers [111,113], and limited evidence from our review is consistent with this interpretation, as the 2 interventions described as EHR-integrated both reported positive outcomes. However, the small number of studies prevents causal interpretation, and integration alone may be insufficient to ensure effectiveness. A CDSS deployment study reported implementation failure despite high algorithmic performance because only a minority of required input variables were available at the point of care, limiting the system’s practical usability during clinical decision-making [114]. EHR integration may also constrain decision support functionality or require substantial local infrastructure investment, particularly in fragmented primary care systems [112,113].

Interpretation of these findings should also consider systematic differences in outcome selection and evaluation timing across implementation contexts. In our review, AI-CDSSs implemented in structured contexts primarily evaluated process-level outcomes, whereas patient-level clinical outcomes were less frequently assessed, a pattern consistent with prior reviews [115,116]. Process outcomes are more readily measurable within short evaluation windows and are often more immediately responsive to decision support interventions [89,116]. However, improvements in process-level outcomes do not necessarily translate into patient-relevant benefit [89,116,117]. Reliance on intermediate end points has been associated with overestimated effects compared with trials evaluating patient outcomes directly [118]. In individualized contexts, where outcomes are shaped by interacting clinical and contextual factors across longitudinal care trajectories, isolating change specifically to AI-CDSSs becomes methodologically challenging [119]. Short evaluation timing was common in our review and may have captured early implementation phases before stable adoption patterns were established. Clinician engagement with decision support tends to evolve over time as familiarity, workflow demands, and implementation barriers change [31,120,121]. These interpretive challenges may partly reflect a mismatch between the predominantly point-in-time logic of current AI-CDSSs and the longitudinal structure of primary care. Most systems in our review were designed to support discrete decisions. This approach is well suited to structured tasks but less aligned with care that unfolds across encounters, depends on evolving patient goals, and requires ongoing adaptation to changing clinical circumstances [122,123]. Achieving more consistent effects in complex primary care settings may require forms of decision support evolving along with care trajectories, clinician judgment, and shared decision-making. Evaluation strategies may also need longer follow-up periods and clearer prespecified linkage between process-level changes and patient-level outcomes.

Safety Considerations

Current reporting of AI-CDSS implementation research captures safety through fragmented and inconsistently specified indicators. Only 2 studies in this review prespecified safety outcomes, and most relied on the absence of serious adverse events as a proxy for safety, with limited detail regarding how harms were identified, monitored, or attributed. This approach risks missing harms that are subtle, cumulative, or delayed, such as missed follow-up, delayed diagnosis, or inappropriate treatment, because these events often require prospective monitoring rather than retrospective absence of signals [124]. In primary care, these risks are amplified by the longitudinal and distributed nature of care [125]. Clinical presentations are heterogeneous, care unfolds across repeated encounters, and adverse outcomes often cannot be attributed to a single decision or clinician interaction [126,127]. Diagnostic delays in ambulatory care, for example, frequently emerge from failures in follow-up coordination and clinical sensemaking across complex work systems rather than from isolated technical errors [128]. Similar concerns have been raised across the broader AI-CDSS literature, where safety reporting often remains secondary to performance evaluation, and benefits and harms are frequently assessed separately, limiting interpretation of overall clinical impact [34,129,130]. Recent expert consensus statements have also highlighted the absence of structured adverse event reporting methodologies and postdeployment monitoring strategies for AI systems in clinical care [38], while calls for systematic “algorithmovigilance” remain only partially operationalized in practice [37].

Our findings further suggest that AI-CDSS safety in primary care cannot be adequately understood through isolated technical or patient-level indicators. Safety events involving digital systems characteristically emerge from interactions among technical performance, clinician behavior, workflow conditions, and organizational context [93,126,131,132]. These dimensions of safety were largely unreported in the studies included in our review. This gap may be particularly important in primary care, where high cognitive load and cumulative alert exposure increase the likelihood of automation bias and alert fatigue [94,133,134]. Harm in primary care can also often accumulate longitudinally across encounters rather than concentrating in discrete events detectable through standard adverse event monitoring [125,128]. Without structured and adapted safety evaluation, implementation studies risk producing evidence for AI-CDSS adoption without establishing whether these systems are, in practice, safe to use. These features of primary care point to evaluation requirements that are not generic but structurally specific to the setting. These include prospective specification of safety outcomes inclusive of process-level harms such as missed follow-up and delayed referral [38,65]; longitudinal monitoring during routine use rather than at deployment alone [127,135]; systematic reporting of clinician engagement with AI recommendations, including override patterns [94,133]; and monitoring intensity calibrated to the clinical consequence of supported decisions, which in primary care spans chronic disease management to acute triage [65,136]. Without evaluation approaches adapted to these features of primary care, the growing body of AI-CDSS implementation evidence risks establishing effectiveness without establishing safety, reversing the expected evidence sequence in clinical research [55].

This review has subject-matter and methodological limitations. First, our search strategy was developed in collaboration with an information specialist (FB) and tested for sensitivity. However, we identified several studies through citation searching and trial registration number. This reflects the indexing inconsistencies and incomplete database coverage for CDSS, despite the existence of a MeSH term since 1997. Our methodology, including backward and forward citation screening in identified knowledge syntheses records, reduces the risk of missing studies that would affect the overall result interpretation. As AI-CDSS implementation studies continue to emerge, the evidence synthesized in this review reflects the literature available up to the predefined search dates (June 27, 2025). Future work should periodically repeat this search approach to monitor newly published evaluations and incorporate emerging evidence over time. Second, substantial heterogeneity across interventions, study designs, implementation contexts, and reported outcomes precluded meta-analysis and limited the use of summary effect estimates for strength-of-evidence interpretation. We therefore used synthesis without meta-analysis based on direction of effect. As emphasized in the SWiM guidance [71], this synthesis approach answers whether there is evidence of effects across studies, rather than estimating an average intervention effect. Our findings should be interpreted as patterns of reported effects across heterogeneous AI-CDSS implementations, not as a pooled estimate of AI-CDSS effectiveness in primary care. Although this approach was appropriate given the available evidence, it carries a risk of interpretive bias. We mitigated this risk through structured outcome grouping, predefined implementation context criteria, and review by a multidisciplinary team of researchers, clinicians, and people with lived experience. Beyond these methodological considerations, interpretation of the findings should also consider the well-recognized international variation in the organization and delivery of primary care across health systems [1,137]. Although all included studies met our predefined eligibility criteria for primary care, they involved different clinician groups and care settings across countries. AI-CDSSs evaluated in 1 primary care system may not perform similarly when implemented in health systems with different organizational structures or models of care, which may limit the transferability of these findings. Third, limitations related to the included studies constrain interpretation. The included studies were conducted exclusively in high-income countries, which limits the generalizability of these findings to low- and middle-income settings. The restriction to English and French publications may have limited the identification of relevant studies from other regions. This choice was made to ensure accurate data extraction and interpretation, as studies in other languages could not be reliably assessed by the review team. The classification of implementation contexts (simple, complicated, and complex) involved an element of interpretive judgment. To enhance transparency and reproducibility, we applied predefined criteria informed by the NASSS framework, including condition complexity, decision logic, system complexity, and preference sensitivity. Classification was performed using explicit decision rules and reviewer consensus. However, this approach remains dependent on the level of detail reported in primary studies, which was often limited, and may therefore not fully capture the underlying complexity of implementation contexts. Finally, evidence from the broader CDSS literature suggests that effectiveness may vary over time, with more recent systems showing modest improvements in clinician behavior [138]. However, we were unable to extract information on the development timeline, versioning, or updating procedures of AI models, as these data were not reported in the included studies. This limits the ability to determine whether observed effects reflect implementation context or differences related to the maturity of the systems evaluated.

Conclusions

Available evidence on AI-CDSSs implemented in primary care demonstrates limited and inconsistent effectiveness, with safety poorly assessed. While positive effects were observed in simple clinical implementation contexts, these were primarily limited to clinician- and system-level outcomes. Improvements in outcomes were not consistent across more complex contexts and rarely translated into patient-level benefits. Safety conclusions are limited by inconsistent evaluation at the AI system level, patient level, and clinician level. The effectiveness conclusions are constrained by heterogeneity, short follow-up, and high or critical risk of bias. By synthesizing comparative evidence from clinically implemented AI-CDSSs across clinician-, patient-, and system-level outcomes and explicitly considering safety and implementation context, this review extends prior reviews that primarily mapped AI applications or implementation barriers in primary care. Clinical implementation research should evaluate safety as a multilevel construct and describe how implementation context shapes the use and impact of AI-CDSSs in clinical practice. Health care systems are adopting AI-CDSSs faster than they can ensure reliable and safe use, a gap demanding rigorous workflow integration, standardized evaluation frameworks, and systematic safety assessment to support trustworthy implementation in primary care.